Lecture 6: Data Acquisition - Client-Side Websites and APIs

class: center, middle, inverse, title-slide

.title[
# Lecture 6: Data Acquisition - Client-Side Websites and APIs
]
.author[
### James Sears* <br> AFRE 891 SS 24 <br> Michigan State University
]
.date[
### .small[<br> *Parts of these slides related to APIs are adapted from <a href="https://github.com/uo-ec607/lectures">“Data Science for Economists”</a> by Grant McDermott.]
]

---

# Table of Contents

1. [Prologue](#prologue)

2. [APIs](#api)

3. [Explicit API Endpoint, No Key](#nokey)

4. [Explicit API Endpoint with Key](#key)

5. [Hidden APIs](#hidden)

6. [Considerate Web Scraping](#consider)

---
class: inverse, middle
name: prologue

# Prologue

---
# Client-Side

.hi-medgrn[Last time], we learned workflows for scraping .hi-medgrn[server-side] websites.
  - Static sites with .hi-slate[rvest]
  - Dynamic or interactive sites with .hi-slate[selenider] and  .hi-slate[rvest]

.hi-purple[Today], we'll turn our attention to .hi-purple[client-side] websites.

---
# Packages for Today

First, a few packages for today:

```r
pacman::p_load(lubridate, tidyverse, jsonlite, httr, usethis, fredr, XML, robotstxt)
```

Also recommended: [JSONView](https://jsonview.com/), a browser extension for Firefox and Chrome that renders JSON output nicely.

---
class: inverse, middle
name: apis

# APIs

---
# Client-Side Web Sites

Recall that websites built using a .hi-purple[client-side] framework typically involve the following steps:
  - Visit a URL containing an .hi-medgrn[empty template] of static content
  - In the process of opening the URL, your browser sends a .hi-pink[request] to the host server
  - If the request is valid, the server issues a .hi-blue[response] and populates the page dynamically with the requested data
  - The final page you see is thus a mix of static and dynamic content

---
# APIs

An .hi-medgrn[API (Application Programming Interface)] is a set of rules that carries out all this requesting, responding, and rendering. Some key concepts:

* .hi-slate[Server:] A powerful computer that runs an API.
* .hi-slate[Client:] A program that exchanges data with a server through an API.
* .hi-slate[Protocol:] The "etiquette" underlying how computers talk to each other (e.g. HTTP).
* .hi-slate[Methods:] The "verbs" that clients use to talk with a server.
  - The main one that we’ll be using is GET (i.e. ask a server to retrieve information).
  - Other common methods are POST, PUT and DELETE.
  
---
# APIs

An .hi-medgrn[API (Application Programming Interface)] is a set of rules that carries out all this requesting, responding, and rendering. Some key concepts:

* .hi-slate[Request:] What the client asks of the server (see Methods above).
* .hi-slate[Response:] The server’s response. This includes:
  - Status Code (e.g. “404” if not found, or “200” if successful).
  - Header (i.e. meta-information about the response).
  - Body (i.e. the actual content that we’re interested in).

---
# API Endpoints

With web APIs, we can access information *directly* from the API database if we can specify the correct URL, known as an .hi-medgrn[API endpoint].

- Similar to normal URLs, but the resulting page is... visually unappealing

---
# API Endpoints

The reason the site looks like hot <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M135.2 17.7L128 32H32C14.3 32 0 46.3 0 64S14.3 96 32 96H416c17.7 0 32-14.3 32-32s-14.3-32-32-32H320l-7.2-14.3C307.4 6.8 296.3 0 284.2 0H163.8c-12.1 0-23.2 6.8-28.6 17.7zM416 128H32L53.2 467c1.6 25.3 22.6 45 47.9 45H346.9c25.3 0 46.3-19.7 47.9-45L416 128z"/></svg> is that its structured and formatted in a very particular way (that can be read directly into R!).

Usually these pages are formatted in one of two ways: XML or JSON
  - We don't need too worry too much about the particular syntax of either

In general there are .hi-blue[three types] of API Endpoints:

1. .hi-purple[Explicit] API Endpoint, .hi-medgrn[no API Key]
1. .hi-purple[Explicit] API Endpoint, .hi-dkgrn[API Key Required]
1. .hi-pink[Hidden] API Endpoints

Let's work through an example workflow for each. But first...

---

# Some APIs that are free & legal

Some fun APIs:
* [TikTok (unofficial)](https://github.com/davidteather/TikTok-Api)
* [Twitter](https://developer.twitter.com/en/docs/twitter-api)
* [Spotify](https://developer.spotify.com/documentation/web-api/)
* [YouTube](https://developers.google.com/youtube/v3/getting-started)

This list of [Public APIs](https://github.com/public-apis/public-apis) has many more.

---

# R API Packages

Many R packages have been built to make working with APIs easier. Some examples:
* .hi-slate[tidycensus] for [American Community Survey microdata and other Census Bureau data](https://walker-data.com/tidycensus/)
* .hi-slate[fredr] for [more easily doing what we will do today](https://cran.r-project.org/web/packages/fredr/vignettes/fredr.html)
* .hi-slate[Quandl] for [Nasdaq and other financial information](https://github.com/quandl/quandl-r)
* .hi-slate[rtweet] for [Twitter data](https://github.com/ropensci/rtweet)
* .hi-slate[rnoaa] for [NOAA weather data](https://github.com/ropensci/rnoaa)
* .hi-slate[prism] for [PRISM gridded climate data](https://github.com/ropensci/prism) (we'll work with this later on)
* .hi-slate[TADA] for [Water Quality Portal Data (EPA, USGS)](https://github.com/USEPA/TADA) 
* .hi-slate[rnassqs] for USDA's [QuickStats API](https://github.com/ropensci/rnassqs) (agricultural data)
* .hi-slate[osmdata] for [OpenStreetMap spatial data](https://cran.r-project.org/web/packages/osmdata/vignettes/osmdata.html)

---
class: inverse, middle
name: nokey

# Explicit API Endpoint, No Key

---
# Explicit API Endpoint, No Key

Let's work through an example of the first type of API: .hi-purple[explicit API without a key].

Let's use the impressive [NYC Open Data](https://opendata.cityofnewyork.us/) to look at... <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M210.6 5.9L62 169.4c-3.9 4.2-6 9.8-6 15.5C56 197.7 66.3 208 79.1 208H104L30.6 281.4c-4.2 4.2-6.6 10-6.6 16C24 309.9 34.1 320 46.6 320H80L5.4 409.5C1.9 413.7 0 419 0 424.5c0 13 10.5 23.5 23.5 23.5H192v32c0 17.7 14.3 32 32 32s32-14.3 32-32V448H424.5c13 0 23.5-10.5 23.5-23.5c0-5.5-1.9-10.8-5.4-15L368 320h33.4c12.5 0 22.6-10.1 22.6-22.6c0-6-2.4-11.8-6.6-16L344 208h24.9c12.7 0 23.1-10.3 23.1-23.1c0-5.7-2.1-11.3-6-15.5L237.4 5.9C234 2.1 229.1 0 224 0s-10 2.1-13.4 5.9z"/></svg>.

Seriously, trees.